System and method for linking real-world objects and object representations by pointing

ABSTRACT

A system and method are described for selecting and identifying a unique object or feature in the system user&#39;s three-dimensional (“3-D”) environment in a two-dimensional (“2-D”) virtual representation of the same object or feature in a virtual environment. The system and method may be incorporated in a mobile device that includes position and orientation sensors to determine the pointing device&#39;s position and pointing direction. The mobile device incorporating the present invention may be adapted for wireless communication with a computer-based system that represents static and dynamic objects and features that exist or are present in the system user&#39;s 3-D environment. The mobile device incorporating the present invention will also have the capability to process information regarding a system user&#39;s environment and calculating specific measures for pointing accuracy and reliability.

CROSS-REFERENCE TO RELATED APPLICATION

This continuation application claims priority under 35 U.S.C. §119(e) toU.S. application Ser. No. 13/413,008 entitled “System and Method forLinking Real World Objects and Object Representations by Pointing,”filed on Mar. 6, 2012, which is a continuation of U.S. application Ser.No. 12/645,248 entitled “System and Method for Linking Real-WorldObjects and Object Representations by Pointing,” filed on Dec. 22, 2009,which claims priority to U.S. Provisional Application 61/139,907 filedon Dec. 22, 2008, and entitled “System and Method for Linking Real WorldObjects and Object Representations by Pointing,” the entire contents ofwhich are incorporated herein in their entirety.

FIELD OF INVENTION

The present invention relates generally to computer-based systems andmethods for identifying objects in the real world and linking them to acorresponding representation in a virtual environment. Morespecifically, the present invention relates to distributedcomputer-based systems and methods for linking objects or featurespresented to and pointed at by system users in a real world environmentto representations of these objects or features in a two-dimensionalvirtual representation, and the identification and assessment of thereliability and accuracy of such identification by pointing to theobject or feature.

BACKGROUND OF THE INVENTION

In recent years, pointing devices have become popular for differentapplications in diverse fields, such as location-based services (LBS),gaming, entertainment, and augmented reality applications. For example,LBS use pointing for identifying geographic objects and features, andreturn information about these objects or features to the systems user.

In gaming, pointing is becoming popular with handheld joystick-likedevices, such as Nintendo's Wii® console. “Wii” is a registeredtrademark of Nintendo Corporation. These joystick-like device allowssystem users to perform movements for interfacing with the game. Inthese gaming systems, motion vectors are captured by sensors build intothe handheld devices. These motion vectors are transmitted to the gameengine and used to emulate gestures within the game scenario, allowing amapping of actions from the real world into a virtual gamingenvironment.

Conventional laser pointers have been used for a long time to direct theaudience's attention to specific objects displayed on a screen or withinthe environment where the presentation is taking place. This examplefurther amplifies that “pointing” has a wide variety of uses andapplications. These uses and applications will only increase as newhandheld devices come onto the market that have increased capabilitiesfor deducing position, determining the direction of pointing, as well asacceleration vectors of pointing gestures.

One of the problems associated with conventional “pointing” systems isthey are inaccurate. This is mainly because the act of “pointing” isinherently ambiguous. This ambiguity arises because it is not alwaysobvious at which object or feature the pointing device is actuallydirected when objects are close together or overlapping. Although thereare many reasons for this inability to accurately identify objects orfeatures through pointing, a main reason for this inaccuracy is that“line of sight” and “pointing direction” are not always aligned. Thus,the ray derived from the orientation of the pointing device may identifya different object or feature than the object or feature the observer(system users) is actually pointing at. This error or uncertainty is dueto an inability of observers (system users) to exactly align their lineof vision with the pointing direction.

A second main reason for pointing uncertainty is based on the inaccuracyof the device being used for pointing. This applies to sensors thatdetermine the pointing device's location and sensors responsible forproviding the direction of pointing. The direction of pointing refers tothe orientation of the pointing device.

The readings of the two sets of sensors combine to derive the ray thatis used for identifying the object or feature of interest. Both of thesetypes of sensors typically have certain characteristics in terms oferrors and uncertainty that are considered when attempting to identifyobjects or features in the real-world environment by pointing at them.

Another reason for pointing accuracy and uncertainty is that humansoften resort to cognitive processes, such as verbal descriptions of theobject or feature of interest, in order to compensate for pointingdevice errors. However, conventional computational pointing systems donot have such cognitive capabilities. As such, the system user's use ofcognitive processes many times leads to erroneous object or featureidentification or inaccurate pointing results.

The present invention overcomes the problems of conventional systems andprovides a system and method that accounts for the deficiencies of suchconventional systems and enhances pointing-based systems so they willmore accurately identify objects or features of interest by pointing.

SUMMARY OF INVENTION

The present invention is a system and method for selecting andidentifying a unique object or feature pointed at by a system user witha pointing device in his/her three-dimensional (“3-D”) environment in atwo-dimensional (“2-D”) representation of that environment, whichincludes the same object or feature. The present invention may beincorporated in a mobile device, preferably a handheld device, thatincludes position and orientation sensors to determine the pointingdevice's position and pointing direction. The mobile deviceincorporating the present invention may be adapted for wirelesscommunication with a virtual computer-based system that is capable ofrepresenting static and dynamic objects and features that exist or arepresent in the system user's real world environment. The mobile deviceincorporating the present invention will also have the capability toprocess information relating to a system user's environment andcalculate specific measures for pointing accuracy and reliability.

Using the present invention, system users may point at objects orfeatures with sufficient accuracy and consider sensor errors in order toidentify the most likely object or feature to which the pointing deviceis being pointed. Further, using the present invention, system userswill be capable of aligning their line of sight and the handhelddevice's pointing direction such that there is minimal uncertainty inpointing compared to the uncertainty or error caused by inaccurateposition and direction readings by the handheld device sensors.

Handheld devices incorporating the present invention account for thepeculiarities of human visual perception by considering the systemuser's visual perspective when pointing at objects or features ofinterest. These considerations add significantly to objectidentification and the reliability assessment process of the presentinvention.

The present invention includes modeling of perceptual and cognitivemechanisms, such as the generation of the system user's field of view ata given time and location and the grouping of objects and features inthat field of view (e.g., set of buildings becomes the district in thevisual scene). This modeling of perceptual and cognitive mechanismsenables the present invention to point at specific ornaments or featuresof objects, or groups of objects that are perceived as one entity by thesystem users for purposes of identification.

The present invention facilitates the integration of uncertainty ofpointing derived from the identification of objects in the visual fieldand uncertainty derived from the inaccuracies of the sensors of thepointing device into the system and method of the present invention foraccurate object or feature identification. This process may be used foridentifying objects or features that exist in a 3-D environment in a 2-Dvirtual representation by pointing at such objects or features in thereal world. This process is based in large part on the 3-D real-worldrepresentation being accurately represented in the 2-D virtualrepresentation.

The present invention is not limited to a single spatial frame ofreference, e.g., world geodetic system 84 (“WGS84). The presentinvention is capable of being configured such that the coordinates ofobjects or features may be translated between multiple frames ofreference. For example, such translation may be necessary for a systemuser to point at a 3-D object or feature on a 2-D TV screen or otherdisplaying device and still effect accurate object identification. Inthis case, the system user and 2-D TV-screen are located in an absoluteframe of reference, and the objects or features on the 2-D TV screen arelocated in the screen's local frame of reference. Given that the spatialconfiguration between the system user and 2-D TV screen, and the localcoordinate system used for displaying the 3-D object on the 2-D TVscreen are known, the present invention enables identifying objects orfeatures on the 2-D TV screen using the pointing device.

The present invention also includes the capability for identifyingmoving objects or features present in the system user's 3-D environmentor on 2-D screens. This capability is enabled by the moving objects orfeatures also being represented in the virtual computer-basedrepresentation, and such moving object or features have their positionand direction of movement updated in real time. According to the presentinvention, these moving object or features will be integrated in the 2-Dvirtual representation of the system user's visible 3-D environment and,therefore, made available for purposes of object or featureidentification.

The system and method of the present invention will be described ingreater detail referring to the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a block diagram of a representative system for carrying outthe present invention.

FIG. 2 shows a workflow for carrying out an exemplary method of thepresent invention.

FIG. 3 shows an example of the optical principles for scene generationby the human visual system.

FIG. 4 shows a representation for the illustrating the concept of theinfluence of distance on the visual angle.

FIG. 5 shows a translation of a 3-D real-world representation to a 2-Dvirtual representation.

FIG. 6 shows a representative method for object or featureidentification in a 2-D virtual representation.

FIG. 7 shows an exemplary method for calculating a distance estimate forobject or features.

FIG. 8 shows a 3-D graphical representation associated with thecalculation of pointing accuracy.

FIG. 9 shows the visual portion of objects or features for defining theoptional pointing locations for convex shapes and concave shapes.

FIG. 10 shows an example of statistical testing according to the processof the present invention.

FIG. 11 shows an example of the use of the method of the presentinvention.

DETAILED DESCRIPTION OF THE PRESENT INVENTION

The present invention is a system and method for selecting andidentifying a unique object or feature in the system user'sthree-dimensional (“3-D”) real-world environment in a two-dimensional(“2-D”) virtual representation of that environment that includes thesame object or feature. Preferably, the present invention isincorporated in a handheld device that includes position and orientationsensors to determine the handheld device's position and pointingdirection. Preferably, a handheld device incorporating the presentinvention is adapted for wireless communication with the computer-basedsystem that includes static and dynamic representations of objects andfeatures that exist or are present in the system user's real-worldenvironment. A handheld device incorporating the present invention alsohas the capability to process information relating to the system user'sreal-world environment and calculate specific measures for pointingaccuracy and reliability.

Referring to FIG. 1, generally at 100, a representative system forcarrying out the present invention will be described. In FIG. 1, thereal world is shown at 102 that includes system user element 104 andenvironment element 106. System user 114 in system user element 104experiences environment element 106. FIG. 1 also shows a system 108 thatincludes system client 110 and system server 112. The interaction andinterconnection of these elements will now be described.

Generally, system client 110 links object or features (represented byelement 118) system user 114 perceives in visual scene 116 to theircounterparts in the 2-D virtual representation based on pointing systemclient 110 at these objects or features 118 in visual scene 116 in thereal-world. The objects or features in virtual scene 116 form the realworld phenomenon 120 in environment 122. In carrying out the method ofthe present invention, preferably, there is an accounting for theinaccuracies of system components, such as in the client system sensorsor other system sensors, and for the projective nature of human visionin order to ensure reliable identification of objects or features ofinterest based on pointing.

System user element 104 includes system user 114 that perceives objectsor features of interest 118 to be identified in visual scene 116. Systemuser 114 is located in environment 122 and observes real-worldphenomenon 120. Visual scene 116 forms at least part of environment 122that system user 114 perceives. Visual scene 116 will be based on theposition and pointing direction of system client 110.

Information relating to visual scene 116 and real-world phenomenon 120is input to system server 112 for scene generation. This is done by thevisual scene information being part of the information from environment122 that is mapped in 3-D representation form to system server 112 at126. The 3-D representation is input to scene generator 128 to generatea 2-D virtual scene representation of the system user's environment at134. System server 112 also will assess the pointing accuracy of clientserver 110 to identify the most probable object or feature the systemuser is pointing at.

Preferably, system client 110 includes a system user interface, sensorsfor generating the current position within an absolute 3-D frame ofreference (e.g., WGS84 for GPS, local coordinate system for indoorpositioning systems), and sensors for generating pointing direction as atwo-valued vector representing pitch and yaw with respect to theabsolute frame of reference. These measurements of system client 110 areprocessed by positioning and pointing device 124.

It is understood that in geodetic terms, pitch and yaw are referred toas elevation angle φ and azimuth θ. Further, preferably system client110 includes modules for time determination, e.g., a system clock, andfor communicating with the system server. The module for communicatingwith system server 112 includes, but is not limited to, wirelesscommunication methods, such as WiFi, HSDPA, GSM EDGE, UMTS, CDMA2000,and WiMAX.

Referring to system server 112, it includes a spatio-temporalrepresentation of the system user's surroundings at 3-D Representationof Environment and Phenomenon 126, and scene generator 128 thatgenerates an annotated 2-D scene representation at 134 of current visualscene 116 as perceived by system user 114 and based on information inputfrom a module for communicating with system client 110. The temporalnature of the 3-D representation just described enables the presentinvention to point at and identify objects or features that arecurrently visible and/or moving through the visual scene. System server112 also includes processes at 130, 132, 136, and 138 for (i) derivingprobability values for actual pointing and probabilities for optimalpointing, (ii) performing statistical tests on these probabilities, and(iii) providing feedback to system user 114 through system client 110.The method of the present invention in view of the system elements shownin FIG. 1 will be described in detail subsequently.

Referring to FIG. 2, generally at 200, a preferable workflow accordingto the present invention is shown. This workflow is carried out by thesystem elements shown in FIG. 1 for the identification of the mostlikely object or feature to which system client 110 is being pointed.Generally, according to FIG. 2, there is (i) at 202 capture of theposition and orientation of the pointing device, namely client server110, and submission of the captured information to system server 112;(ii) at 204 generation of a 2-D scene representation of the visual sceneperceived by the system user from the system user's 3-D spatio-temporalrepresentation in the system server; (iii) at 208 generation of a set ofprobability surfaces for optimal pointing from the 2-D virtual scenerepresentation; (iv) at 206 estimation of the distance from the systemuser's position to objects or features in the 3-D representation andcalculating a set of probability ellipsoids at the object's or feature'sestimated distance for assessing actual pointing accuracy (errorpropagation); (v) at 210 comparison of the pairs of probabilities foroptimal pointing and actual pointing for individual objects or features;and (vi) at 212 ranking and providing feedback to the system user as towhich object or feature a pointing device is most likely being pointed.The system configuration and workflow will be now described in greaterdetail.

People point at objects in real world every day. Although human visionand pointing are related, pointing does not necessarily translate intovision. An environmental scene surrounding a system user, with orwithout vision, can be generated and described by system server 112 froma 3-D representation or model of the real world. This scene generationcorresponds to the mapping of the 3-D representation input to 3-DRepresentation of Environment and Phenomenon 126 into a projective 2-Dvirtual representation. In the projective 2-D virtual representation,object or features consists of faces or sides denoting the visible partof such objects or features. This 2-D virtual representation representsthe configuration of system user 114's current visual scene 116. Visualscene 116 include static objects, such as buildings or city features,and also dynamic (moving) objects, such as cars or boats. The 2-Dvirtual representation also captures the restrictions in system user114's visual scene, particularly in terms of distance of objects orfeatures, visibility, and the occlusion of object or features. Accordingto the present invention, based on the 2-D virtual representation andthe pointing direction of system client 110, a set of computationalsteps may be performed to identify with substantial accuracy the objector feature to which system client 110 is being pointed.

For purposes of the present invention, scene generation is acomputational approximation of the human visual system. The human visualsystem permits people to assimilate information from the environment andpoint at the corresponding objects or features. FIG. 3 shows the opticalprinciples of scene generation based on the human visual system.

Referring to FIG. 3, generally at 300, human eye 308 is shown withrespect to three-coordinate system 305 and observed phenomenon, namely,building 318, silo 320, and molecular structure 322. A presence of themolecular structure as a phenomenon in FIG. 3 is provided to demonstratethat the object or feature of interest is not limited to onlylarge-scale object or features. As such, the present invention may beused for identifying very small objects or features, for example, usinga microscope, as objects or features of interest among other smallobjects or features.

Three-coordinate system 305 includes x-axis 302, y-axis 304, and z-axis306. Eye 308 shows cornea 310, lens 312, and retina 314. As shown withregard to eye 308, focal point 309 of lens 312 is positioned withinthree-dimensional coordinate system 305. The eye elements are used toprovide 2-D sensed illustration 316 on retina 314 of the 3-D real worldscene that includes building 318, silo 320, and molecular structure 322.

The image on retina 314 is the perceptual representation of the observedreal-world phenomenon at which the system user may point. The projectivetransformation from 3-D real-world objects or features into a 2-D imageon retina is the “visual perspective.” The visual perspective operateson the assumption that the observer is located a certain distance fromthe observed objects. As the objects or features become more distant,they will appear smaller because their angular diameter, visual angle,decreases. The visual angle of an object or feature is a trianglesubtended at the eye by a triangle with the height of the object orfeature as its base. Therefore, the further the object or feature isfrom the eye, the smaller the visual angle.

Referring to FIG. 4, generally at 400, the influence of distance onvisual angle will be described as it relates to the human visual system.If the Sun and Moon were placed side by side, the Sun would be manytimes larger than the Moon. However, referring to FIG. 4, the Sun andMoon will appear the same size because the visual angle for the Sun andMoon is the same.

Again, referring to FIG. 4, from viewpoint 402, an observer would seeSun 406 subtended by rays 408 and 410. These rays will form visual angle412 on phantom line circle 404 about view point 402 given the distancebetween viewpoint 402 and Sun 406. From viewpoint 402, Moon 414, whichis much closer to viewpoint 402 than Sun 406, is subtended by rays 416and 418. Rays 416 and 418 form visual angle 420 on phantom line circle404, which is the same visual angle as visual angle 412. Therefore, when∠Moon=∠Sunthen the Moon and Sun will appear the same size from viewpoint 402although the Sun is much larger than the Moon.

The relationship between distance and the apparent heights of objects isnot linear. For example, if an object or a feature is actually extremelyclose to the eye, virtually touching the eye, it would appear infinitelytall based on a range of vision of the eye.

Again, referring to FIG. 1, the 3-D model of the environment at 3-DRepresentation and Phenomenon 126 that includes visual scene 116containing objects and features of interest (element 118) is transformedat scene generator 128 into a 2-D virtual representation of the 3-Dmodel. This 2-D virtual representation of the 3-D model forms the inputfor carrying out the remaining steps of the process of the presentinvention.

The computational generation of the 2-D virtual representation will becarried out using a projective transformation that projects the 3-Drepresentation of real-world scene and objects or features within itinto a 2-D view plane. For a more accurate imitation of a human visionperception of a real world, the transformation of the 3-D representationwould be the projection onto a spherical 2-D representation. It isunderstood that the spherical 2-D representation is considered withinthe scope of the present invention.

Referring to FIG. 5, generally at 500, the transformation of a 3-Drepresentation of the real-world scene to a 2-D virtual representationwill be discussed. In FIG. 5, three-coordinate coordinate system 502 isshown with x-axis 504, y-axis 506, and z-axis 508. System user 114(FIG. 1) is shown at position 510 within three-coordinate system 502. At510, the pointing direction of system client 110 is shown along ray 512.Therefore, from the viewpoint at 510 with pointing direction 512, the2-D virtual representation 520 would be created. The 2-D virtualrepresentation will be a flat, scaled-down version of the objects andfeatures observed by the system user 114 in the real world.

2-D virtual representation 520 that is generated is shown in larger viewat 530. The objects or features as viewed in the 2-D virtual scenerepresentation would be the visible parts of phenomenon building 514,silo 516, and molecular structure 518. More specifically, 2-D virtualrepresentation 520 shows the projected silhouettes of the visible partsof building 514, silo 516, and molecular structure 518. Further, thepointing direction (pointing location of ray 512) is shown at 522.

As stated, only silhouettes of objects or features would be seen fromviewpoint 510 instead of the entire object or feature. Accordingly, the2-D virtual representation that is generated by screen generator 128forms the basis on which probabilities of optimal pointing and actualpointing may be assessed according to the process of the presentinvention. Further, based on probabilities associated with optimalpointing and actual pointing, the object or feature being pointed at bysystem client 110 is determined. The object identification process ofthe present invention will now be discussed in greater detail.

According to the present invention, the object identification portion ofthe process of the present invention is applied to the 2-D virtualrepresentation generated at screen generator 128 and provided at 2-Dscene representation 134. The object identification process according topresent invention, preferably, includes at least the following fivesteps: (1) calculating of the distance from the system user toindividual object or features in the visual scene at 116; (2) computingprobability ellipsoids for pointing to individual object or featuresusing error propagation; (3) defining the optimal pointing location onvisible object or feature faces; (4) calculating probability surfacesfor optimal pointing on an object or a feature; and (5) conductingstatistical tests for determining correspondence between an object orfeature, and the pointing direction of system client 110. The steps areshown in flow diagram form in FIG. 6 and these steps will be explainedreferring to FIGS. 7-11.

FIG. 6, generally at 600, shows the steps of the process for objectidentification. Step 1 at 602 is for computing for each object orfeature in the 2-D virtual representation its distance from the systemuser's position. Preferably, Step 1 will be carried out at SceneGenerator 128 and 2-D Scene Representation 134. The output of Step 1 at602 is input to Step 2 at 604. Step 2 at 604 is for calculating for eachobject or feature in the 2-D virtual representation the accuracy ofactual pointing using error propagation. Preferably, Step 2 will becarried out at Scene Generator 128, 2-D Scene Representation 134, andLocation of Pointing & List of Pointing Accuracies 130.

Preferably, in parallel with carrying out process Steps 1 and 2, theprocess for object identification will perform Step 3 at 606 and Step 4at 608. Step 3 at 606 is for defining for each object or feature in the2-D virtual representation its optimal pointing location. Preferably,Step 3 at 606 will be carried out at Scene Generator 128 and 2-D SceneRepresentation 134.

The output of Step 3 at 606 is input to Step 4 at 608. Step 4 at 608 isfor calculating for each object in the 2-D virtual representationprobability surfaces for optimal pointing. Preferably, Step 4 at 608will be carried out at Scene Generator 128 and 2-D Scene Representation134.

The output of Step 2 at 604 and the output of Step 4 at 608 are input toStep 5 at 610. Step 5 at 610 is for computing for each pair of optimalpointing and actual pointing the likelihood of the correspondencebetween the respective distributions. Step 5 at 610 will be carried outat Accuracy Assessor 136 and Sorted List of Potential Targets 138.

It is understood that each of the elements shown in system server 112may be separate modules or integrated into one or more elements andstill be within the scope of the present invention.

Referring to FIG. 7, generally at 700, calculation of distance estimatesfrom the system user's position to objects or features according to Step1 at 602 (FIG. 6) will be described. According to Step 1, the distanceto an object or a feature is estimated as a function of the coordinatesof the system user's position and an auxiliary coordinate representingthe object's or feature's visible area. In FIG. 7, the system user'sposition is shown at 702 and the auxiliary coordinate will be associatedwith a visible surface of cube 704. From the system user's point of view(“POV”) at 702, the line projections are shown to each of the object'sor feature's visible vertices, namely, line projections to the vertices1, 2, 3, 4, 5, 6, and 7. Each of the line projections are used forcalculating the auxiliary coordinate for each object or feature. Onlythe visible portions of the object or feature, in this case cube 704,contribute to the estimate of distance, as such the vertex at 8 is notused.

Expression 1 that follows is used for calculating the auxiliarycoordinate for the distance estimate for objects or features in a visualscene, such as visual scene 116. Expression 1 is:

$\begin{matrix}{{x_{P_{aux}} = {\frac{1}{n} \cdot {\sum\limits_{i = 1}^{n}x_{i}}}}\mspace{14mu}{y_{P_{aux}} = {\frac{1}{n} \cdot {\sum\limits_{i = 1}^{n}y_{i}}}}{z_{P_{aux}} = {\frac{1}{n} \cdot {\sum\limits_{i = 1}^{n}z_{i}}}}} & (1)\end{matrix}$where,

x_(Paux)=the x-coordinate of the auxiliary coordinate in athree-dimensional coordinate system.

y_(Paux)=the y-coordinate of the auxiliary coordinate in athree-dimensional coordinate system.

z_(Paux)=the z-coordinate of the auxiliary coordinate in athree-dimensional coordinate system.

n=is the number of visible vertices for each object or feature (which inFIG. 7, n=7).

P_(aux)=the auxiliary coordinate.

The distance, d₀, from the system user's POV 704 to the object's orfeature's auxiliary coordinate, P_(aux), is according to the Euclideandistance between two coordinates as set forth in Expression 2:d _(o)=√{square root over ((x _(P) _(aux) −x _(PoV))²+(y _(P) _(aux) −y_(PoV))²+(z _(P) _(aux) −z _(PoV))²)}{square root over ((x _(P) _(aux)−x _(PoV))²+(y _(P) _(aux) −y _(PoV))²+(z _(P) _(aux) −z_(PoV))²)}{square root over ((x _(P) _(aux) −x _(PoV))²+(y _(P) _(aux)−y _(PoV))²+(z _(P) _(aux) −z _(PoV))²)}  (2)where,

x_(Paux)=from Expression 1.

y_(Paux)=from Expression 1.

z_(Paux)=from Expression 1.

x_(POV)=the x-coordinate of the system user at the POV.

Y_(POV)=the y-coordinate of the system user at the POV.

z_(POV)=the z-coordinate of the system user at the POV.

Referring again to FIG. 7, distance d₀ will be the distance from the POVat 702 to auxiliary coordinate P_(aux) at 708 located at the center ofgravity of the surface bounded by vertices 1, 2, 3, 4, 5, 6 and 7.Auxiliary coordinate 708 would be positioned at this location in thecenter of gravity of vertices 1, 2, 3, 4, 5, 6 and 7 because thislocation represents the perceived distance between the system userlocation and location of the object. Following in the calculation ofdistance d₀ according to Step 1 at 602, this calculation is input toStep 2 at 604.

As stated, Step 2 at 604 of the process of the present invention, theaccuracy of actual pointing with regard to each object or feature in the2-D virtual representation is calculated preferably using errorpropagation.

The accuracy of pointing based on the pointing direction of systemclient 110 is influenced by the accuracy of sensor measurements by thesystem client 110 sensors and distance d₀ that was calculated at Step 1(602). Error propagation considerations incorporated at Step 2 includegeodetic error propagation. Further, the determination of theuncertainty of pointing with respect to sensors preferably accounts forindividual sources of error associated with the sensors.

A potential source of error relates to sensor measurements includes, butis not limited to, sensor errors associated with the measurement oflatitude, longitude, height, azimuth, and elevation angle. These sensormeasurements and distance d₀ will be used to calculate a representativecoordinate for each such object or feature to determine actual pointingaccuracy. The representative coordinate enables the determination of theinfluence of each individual error source of the overall standarddeviation based on the assumption system user 114 is pointing at aspecific object or feature in visual scene 116.

Typically, “standard deviations” are meant to provide a level ofconfidence in statistical conclusions. However, according to the presentinvention, “standard deviation” will be used to characterize thedistribution underlying the probabilistic method for object or featureidentification. The standard deviation of the representative coordinateaccording to present invention is a result of error propagation and canbe calculated by the partial differentials of the function used forcalculating the representative coordinate. According to the presentinvention, geodetic error propagation is determined according toExpression 3 on a function ƒ with n variables:

$\begin{matrix}{\sigma_{f} = \sqrt{\sum\limits_{i = 0}^{n}\left( {\left( \frac{\partial f}{\partial i} \right)^{2} \cdot \sigma_{i}^{2}} \right)}} & (3)\end{matrix}$where,

-   -   σ_(i)=a standard deviation of the corresponding variable.

$\frac{\partial f}{\partial i} = {a\mspace{14mu}{partial}\mspace{14mu}{derivative}\mspace{14mu}{that}\mspace{14mu}{describes}\mspace{14mu}{the}\mspace{14mu}{influence}\mspace{14mu}{that}\mspace{14mu}{the}\mspace{14mu}{uncertainties}\mspace{14mu}{of}\mspace{14mu}{individual}\mspace{14mu}{variables}\mspace{14mu}{have}\mspace{14mu}{on}\mspace{14mu}{the}\mspace{14mu}{{function}'}s\mspace{14mu}{standard}\mspace{14mu}{{deviation}.}}$

Referring to FIG. 8, generally at 800, an exemplary determination ofgeodetic error propagation will be described. In FIG. 8,three-coordinate system 802 has x-axis 804, y-axis 806, and z-axis 808.Point p 810 at the origin represents the system user's POV. Point o 814represents the representative coordinate to be determined and Point o isseparated from the system user's Point p 810 by distance d₀ 812. Asshown, azimuth angle θ is at 816 and elevation angle φ is at 818.Further, three-dimensional Point o 814 maps to Point p′ 822 in thetwo-dimensional x-y plane.

According to FIG. 8, the variables for determining the representativecoordinate of the object or feature of interest include the coordinateof the system user's position at Point p 810 having coordinates x_(p),y_(p), z_(p); yaw, azimuth angle θ at 816; pitch, elevation angle φ at818; and distance d₀ 812 to the object or feature. Standard deviationsfor purpose of the present invention may be in terms of metric distancefor position coordinates and distance and radial distance, for pitch andyaw. For example, metric distance may be represented by σ_(x)=5 m andradial distance may be represented by σ_(ψ)=0.5 rad.

Noting the foregoing, preferably, error propagation for pointingaccuracy may be determined using Expression 4:x _(o) =x _(p) +d ₀·cos(φ)·sin(θ)y _(o) =y _(p) +d ₀·cos(φ)·cos(θ)z _(o) =z _(p) +d ₀·sin(φ)  (4)

Given Expressions 3 and 4, the standard deviation of the representativecoordinate, which will be used to determine the likelihood the systemuser is pointing at a specific object or feature, preferably will becalculated according to Expression 5:

$\begin{matrix}{{\sigma_{x} = \sqrt{{\left( \frac{\partial x_{O}}{\partial x_{p}} \right)^{2} \cdot {\sigma_{x_{p}}^{2}\left( \frac{\partial x_{O}}{\partial d_{O}} \right)}^{2} \cdot \sigma_{d_{O}}^{2}} + {\left( \frac{\partial x_{O}}{\partial\phi} \right)^{2} \cdot \sigma_{\phi}^{2}} + {\left( \frac{\partial x_{O}}{\partial\theta} \right)^{2} \cdot \sigma_{\theta}^{2}}}}{\sigma_{y} = \sqrt{{\left( \frac{\partial y_{O}}{\partial y_{p}} \right)^{2} \cdot {\sigma_{y_{p}}^{2}\left( \frac{\partial y_{O}}{\partial d_{O}} \right)}^{2} \cdot \sigma_{d_{O}}^{2}} + {\left( \frac{\partial y_{O}}{\partial\phi} \right)^{2} \cdot \sigma_{\phi}^{2}} + {\left( \frac{\partial y_{O}}{\partial\theta} \right)^{2} \cdot \sigma_{\theta}^{2}}}}{\sigma_{z} = \sqrt{{\left( \frac{\partial z_{O}}{\partial z_{p}} \right)^{2} \cdot {\sigma_{z_{p}}^{2}\left( \frac{\partial z_{O}}{\partial d_{O}} \right)}^{2} \cdot \sigma_{d_{O}}^{2}} + {\left( \frac{\partial z_{O}}{\partial\phi} \right)^{2} \cdot \sigma_{\phi}^{2}}}}} & (5)\end{matrix}$

For purposes of the present invention, if in determining the standarddeviation according to Expression 5 a particular variable is notavailable, that variable will be viewed as a constant. As such, inpractical terms, the partial derivative of such a variable will be leftout. For example, if the standard deviation for distance d₀ is notknown, the partial derivative for distance d₀ will not be included inthe determination.

In determining pointing accuracy according to the present invention, theprincipal focus is on the application of geodetic error propagation forestimating the standard error of pointing associated with an object or afeature. Accordingly, the distance estimation function and thecoordinate calculation function according to the Expressions above maybe replaced by other functions or estimating methods and still be withinthe scope of the present invention. For example, alternative functions,include but are not limited to, (i) determining the standard deviationof pointing accuracy using transformations across multiple spatialframes of reference or (ii) standard deviation of pointing accuracy mayneed to be aligned with a local coordinate system for furtherprocessing.

The determination of pointing accuracy at Step 2 at 604 is one of theinputs to Step 5 at 610. The generation of the second input to Step 5involving Steps 3 and 4 at 606 and 608, respectively, will now bedescribed.

Step 3 at 606 is for defining the optimal pointing location for eachobject in the 2-D visual scene representation. In referring to theoptimal pointing location, the present invention considers certainpreferable actions and considerations of the system user. Theseconsiderations include that system users do not randomly point atobjects or features, and typically points at the center of an object'sor a feature's visible surface or a salient feature associated with theobject or feature of interest. For example, if the system user ispointing at a rectangle façade of the building, the system user wouldtend to point at the center mass of the façade rather than at the edges.However, in the case of arcs or archways, the optimal pointing locationfor the system user may not be as obvious as in a situation with arectangle façade and the optimal pointing location may require a morecareful definition as will be discussed.

According to the present invention, the optimal pointing location is nota characteristic of the object or feature as a whole, but depends on thesystem user's point of view and the visible parts of the object orfeature presented to the system user. As such, preferably, the presentinvention will generate the 2-D virtual scene representation beforedefining the optimal pointing location.

In defining the optimal pointing location, the geometry of the object orfeature is not the only consideration. The optimal pointing location maybe significantly influenced by the salient features or unusualcharacteristics of the object or feature, including the attentive andnonattentive features, for example, texture, color, material, orornaments. These features, however, may be dependent on the nature ofthe real-world phenomenon. For example, a building that has a salientwindow or ornaments may attract a system user's attention and have asignificant influence on the optimal pointing location with respect tothat building. As another example, if there is a screen display ofmolecules or molecular structures based on microscopic investigations,certain molecules or molecular structures may be considered more salientand prominent than others, again influencing the optimal point location.According to the present invention, these characteristics or featuresmay be integrated as part of the 3-D real world representation of theenvironment and used in the definition of the optical pointing location.

For purpose of example only, if a building had a low level, center massdoor for entering and leaving the building and a 12 foot high bronzestatue of an ornamental horse that is toward the right side of the frontof the building and the horse had become famous with its association tothat building, then the horse would be considered a significantinfluence on the optimal pointing direction for the building.

Noting the foregoing, the definition of the optimal pointing locationwill depend on the shape of the visible faces of the object or featureas presented to the system user. According to Step 3, the definitionthat is input to Step 4 will be based on the following disclosure.

The definition of the optimal pointing location is according to rulesand heuristics defined by the system administrator and by previousresults of the calculation of pointing locations. The rules andheuristics are defined a-priori based on expert knowledge at 3-DRepresentation of Environment 126 and the influence of the generation ofthe 2-D virtual scene at runtime of the Scene Generator 128 and 2-DScene Representation 134. The rules and heuristics, for example, may bedefining a set of prototypical geometries (U-Shape, L-Shape, etc) alongwith their optimal pointing locations, subdividing complex geometriesand defining optimal pointing locations for each part, or analyzingprevious pointing events for given geometries and setting the optimalpointing location accordingly. The learning process employed by SceneGenerator 128 and 2-D Scene Representation 134 may include, but is notbe limited to, the following, 1) recording pointing events, 2)associating pointing events with location on objects and features, 3)statistically deriving optimal pointing location based on clustering ofpointing events for each face, 4) refining pointing location for objectsand features by repeating steps 1, 2 and 3.

At Step 4, the present invention determines the probability surface foroptimal pointing from the system user's position and the visual outlineof the object or feature presented, for example, in 2-D virtualrepresentation at 520 in FIG. 5.

Referring to FIG. 9, examples for determining the probability surfacesfor optimal optical pointing for regular and irregular shapes will bediscussed. For purposes of discussing determining the probabilitysurfaces of optimal pointing for regular shapes, regular shapes areobjects or features with “convex faces,” which would include square orrectangular shapes. For purposes of discussing determining theprobability surfaces of optimal pointing for irregular shapes, irregularshapes are objects or features with “concave faces,” which would includearcs or archways.

Again referring to FIG. 9, generally at 900, regular shape 904 is shownand, generally at 902, irregular shape 912 is shown. Preferably,according to the present invention, the determination of probabilitysurfaces for regular shaped objects or features will be carried outusing the minimum bounding rectangle method. Calculating the optimalpointing probability using the minimum bounding rectangle method willresult in minimum bounding rectangle 906 having optimal pointing region908 that contains concentric ellipsoids 909. The center of optimalpointing region 908 at location 910 would have the highest probabilityfor the optimal pointing location. This location would be on visiblesurface 907 of object or feature 904 rather than visible surface 905.

Preferably, in FIG. 9 at 902 with regard to irregular shaped objects orfeatures, the present invention will determine the probability ofsurfaces optimal pointing using an isoline that describes the irregularshaped object or feature. At 902, irregular shaped object or feature 912has optimal pointing region 914 containing concentric polygons 915. Thisoptimal pointing location in region 914 would be isoline 916.

Having calculated isoline 916 using conventional means, a determinationof the probability of optimal pointing preferably will include theconsideration of human cognitive characteristics. For example, theprobability assignment or weighting for lateral and vertical pointingmay be defined differently due to humans tending to point moreaccurately to lateral than to vertical dimensions. Therefore, using thisconsideration, the optimal pointing location for irregular shape 912would be at 918 on isoline 916.

However, cognitive considerations can have probability values set by thesystem administrator or as default settings to reflect the likelihood ofoptimal pointing. It is understood, however, that other considerationsmay be used for determining the probability of optimal pointing forirregular shaped objects or features and still be within the scope ofthe present invention.

Once the optimal pointing location is defined according to Step 3 at606, the definition is input to Step 4 at 608. Step 4 determines theprobability surface for optimal pointing for each object in the 2-Dvirtual representation. The output of Step 4 is the second input to Step5 at 610.

Step 5 at 610 determines the likelihood of correspondence betweenrespective distributions for each pair of optimal pointing and actualpointing determinations to assess the level of reliability that thepointing direction is aimed at a specific object or feature in visualscene 116. As will be described, the assessment at Step 5, preferablyuses statistical processes for effecting the assessment.

According to the present invention, preferably a statistical t-test isused for assessing correspondence of the distributions. However, it isunderstood that other statistical methods may be used and still bewithin the scope of the present invention.

According to the present invention, the statistical t-test willdetermine the likelihood that the distribution defined by the actualpointing location and standard deviation of pointing according to Step 2at 604 is the same as the distribution defined by the object's orfeature's optimal pointing surface according to Step 4 at 608.

With regard to the statistical t-test, generally it will compare thedifferences between two means in relation to the variation in data. Forpurposes of the present invention with respect to the statisticalt-test, preferably, for each object or feature there will be twoindependent samples for comparison. The first is an optimal pointingsample according to Step 4 and the second is an actual pointing sampleaccording to Step 2. There also is an initial assumption that thevariance in the two samples will be unequal. The samples areparameterized by the representative coordinate for actual pointing andthe optimal pointing location along with their standard deviations. Thet statistic that results from the assessment tests whether thepopulation means are different. The t statistic is determined accordingto Expressions 6 and 7:

$\begin{matrix}{t = \frac{\mu_{act} - \mu_{opt}}{s_{\mu_{act} - \mu_{opt}}}} & (6)\end{matrix}$where,

$\begin{matrix}{s_{\mu_{act} - \mu_{opt}} = \sqrt{\frac{\sigma_{\mu_{act}}^{2}}{n_{act}} + \frac{\sigma_{\mu_{opt}}^{2}}{n_{opt}}}} & (7)\end{matrix}$whereby,

μ=corresponds to the sample mean for normal distributions of eachsample.

σ=standard deviations of the distributions.

s=an estimator of the common standard deviation.

n=the representative sample size for actual pointing and optimalpointing.

For purposes of the present invention, n depends on a number ofobservations that were made that lead to the estimation of the standarddeviation for the location and orientation of system 110 (the pointingdevice). Further, n for optimal pointing is dependent on the modeling ofthe optimal pointing location according to the method used for thedefinition. In the case of rules and heuristics, n depends on the natureof the heuristics, and in the case of learning algorithms, n depends onthe number of previous observations.

Preferably, in carrying out the process of Step 5 at 610, each object'sor feature's distribution for optimal pointing is compared to thedistribution of actual pointing resulting in a list of t-values for eachpair. These resultant t-values are analyzed to determine the probabilitythat a significant difference exists between the two samples of eachoptimal pointing-actual pointing pair that will determine the likelihoodof a specific object or feature in being pointed at by the system user.

The t-test may be performed laterally, vertically, or for both axes andstill be within the scope of the present invention. The statisticalt-test results may be integrated into one measure for reliability ofpointing using an Euclidean distance for the two measures for exampleaccording to Expression 8:t _(tot)=√{square root over (t _(lat) ² +t _(ver) ²)}  (8)

where,

-   -   t_(tot)=the total t-test results for the lateral and vertical        optimal pointing and actual pointing standard deviations.    -   t_(lat)=the t-test results for the lateral optimal pointing and        actual pointing standard deviations.    -   t_(ver)=the t-test results for the vertical optimal pointing and        actual pointing standard deviations.

The preferred method may be replaced by a weighted Euclidean distance,if lateral and vertical pointing are not equally important and it willstill be within the scope of the present invention.

Referring to Figure, 10 generally at 1000, an example of statisticaltesting used by the system of the present invention will be described.It is understood that the statistical t-test is the preferred method forconducting the comparisons at Step 5; however, it is understood thatother statistical methods may be used and still be within the scope ofthe present invention.

Yet again referring to FIG. 10, 2-D scene representation 1002 is basedon two-coordinate system 1004 having x-axis 1006 and y-axis 1008. 2-Dscene representation 1002 includes visible object 1010 and actualpointing location 1020. Given that the object face has a regular shape,the minimum bounding rectangle method may be used for determining theprobability surface for optimal pointing. Using the minimal boundingrectangle method, optimal pointing region 1014 is determined and thecenter of that region is at 1018.

Again, referring to FIG. 10, the t-test setup for a pair of standarddeviations for optimal pointing location 1018 and actual pointinglocation 1020 is shown. The standard distributions for optimal pointing1018 and actual pointing 1020 are compared according to Step 5 in the xand y directions at 1022 and 1024, respectively. The distribution curvesat 1022 show the degree of overlap between the lateral distribution foractual pointing at 1023 and the lateral distribution for optimalpointing at 1025, whereby the area of overlap defines the probabilitythat the two distributions correspond. Further, the distribution curvesat 1024 show the degree of probability that the two distributionscorrespond in the vertical direction. As is shown, the verticaldistribution for actual pointing is at 1026 and the verticaldistribution for optimal pointing is at 1028. As with the lateraldistributions, the overlap of the vertical distributions showed aprobability that the two distributions correspond.

The total overlap of the lateral and vertical optimal pointing andactual pointing distributions will provide the total correspondence ofoptimal pointing and actual pointing for evaluation of the likely objectbeing pointed at by the pointing device.

The results of the comparison process of Step 5 at 610 is a list ofreliability measures that reflect to what degree the pointing isdirected to specific objects or features. For example, the following isa representative list of such results:

-   -   Pair t-test for Actual Pointing and: Result t-test    -   1) Optimal Pointing for Object A 0.45    -   2) Optimal Pointing for Object B 0.18    -   3) Optimal Pointing for Object C 0.82    -   4) Optimal Pointing for Object D 0.75

From the above list, Object C at number 3 would be the winner becausethe result of the t-test indicates the highest probability ofcorrespondence. As such, it would be determined Object C is most likelyobject or feature to which a system user is pointing. The result of Step5 is the identification of the object or feature most likely beingpointed at. This also results in the linking of the real world objectsor features to virtual objects or features. The linking information canbe used for other observations and linking between the real world andthe 2-D virtual representation.

The Gaussian function and statistical tests that are adapted for use inthe system and method of a present invention for optimal pointing andactual pointing consider the distribution anomalies associatedtherewith. For example, pointing at the periphery of a building insteadof the middle results in a lower likelihood that the object or featureis the one to which the pointing device is actually being pointed.Further, given the projected nature of the visual system and theresulting perspective, it is likely the system user is pointing at anobject or feature in the background rather than the object or featurethat is intersected by the pointing ray.

Referring to FIG. 11, generally at 1100, an example will be describedfor selecting the actual object being pointed at by evaluating thedistributions associated with each object or feature consideringdistribution anomalies when two objects may lie in the same pointingdirection. Object A at 1102 with optimal pointing location 1104 andobject B at 1106 with optimal pointing location 1108 have the samegeometric properties in the 2-D scene representation. However, thedistance d1 from the system user's position to object A is shorter thandistance d2 from the system user's position to object B. Object A hasoptimal pointing distribution 1112 and standard distribution 1114. Thesedistributions for object A overlap at 1122. Object B has optimalpointing distribution 1116 and standard distribution 1120. Thesedistributions for object B overlap at 1124.

Again, referring to FIG. 11, the distributions for actual pointing arethe same for each object but the standard deviations are unequal. Theactual pointing directions for both objects is exactly the same, thepresent invention would identify object B as the object most likelybeing pointed at because the overlap of the distributions associatedwith object B is of a higher degree than for object A as shown bycomparing the areas at 1122 and 1124. This is because this areacorresponds to the outcome of the statistical test for the likelihood ofcorresponding distributions.

It is understood that the elements of the systems of the presentinvention may be connected electronically by wired or wirelessconnections and still be within the scope of the present invention.

The embodiments or portions thereof of the system and method of thepresent invention may be implemented in computer hardware, firmware,and/or computer programs executing on programmable computers or serversthat each includes a processor and a storage medium readable by theprocessor (including volatile and non-volatile memory and/or storageelements). Any computer program may be implemented in a high-levelprocedural or object-oriented programming language to communicate withinand outside of computer-based systems.

Any computer program may be stored on an article of manufacture, such asa storage medium (e.g., CD-ROM, hard disk, or magnetic diskette) ordevice (e.g., computer peripheral), that is readable by a general orspecial purpose programmable computer for configuring and operating thecomputer when the storage medium or device is read by the computer toperform the functions of the embodiments. The embodiments or portionsthereof may also be implemented as a machine-readable storage medium,configured with a computer program, where, upon execution, instructionsin the computer program cause a machine to operate to perform thefunctions of the embodiments described above.

The embodiments or portions thereof, of the system and method of thepresent invention described above may be used in a variety ofapplications. Although the embodiments, or portions thereof, are notlimited in this respect, the embodiments, or portions thereof, may beimplemented with memory devices in microcontrollers, general purposemicroprocessors, digital signal processors (DSPs), reducedinstruction-set computing (RISC), and complex instruction-set computing(CISC), among other electronic components. Moreover, the embodiments, orportions thereof, described above may also be implemented usingintegrated circuit blocks referred to as main memory, cache memory, orother types of memory that store electronic instructions to be executedby a microprocessor or store data that may be used in arithmeticoperations.

The descriptions are applicable in any computing or processingenvironment. The embodiments, or portions thereof, may be implemented inhardware, software, or a combination of the two. For example, theembodiments, or portions thereof, may be implemented using circuitry,such as one or more of programmable logic (e.g., an ASIC), logic gates,a processor, and a memory.

Various modifications to the disclosed embodiments will be apparent tothose skilled in the art, and the general principles set forth below maybe applied to other embodiments and applications. Thus, the presentinvention is not intended to be limited to the embodiments shown ordescribed herein.

The invention claimed is:
 1. A computer-implemented method fordetermining a likelihood a moving object in a real-world scene is anobject being pointed at by a mobile device that includes capabilities ofa pointing device, comprising the steps of: (A) pointing a mobileelectronic device that includes capabilities of a pointing device(mobile device) at a moving object of interest in the real-world scene;(B) generating with the mobile device a mobile device geodetic positionand pointing direction, and transmitting the mobile device geodeticposition and pointing direction, and standard deviations relating togenerating the mobile device geodetic position and pointing direction toa system server; (C) mapping a three-dimensional representation of areal-world scene containing the objects in the real-world scene,including the moving object of interest, and transmitting thethree-dimensional representation to the system server; (D) a systemserver performing the substeps of, (1) generating a two-dimensionalvirtual digital representation of the three-dimensional scene mapped atstep (C), with the two-dimensional digital virtual representationincluding at least a virtual digital representation of the objects inthe real-world scene, including the moving object of interest, (2)determining for each object, including the moving object of interest, inthe two-dimensional virtual digital representation a distance estimatefrom the mobile device geodetic position to each such object, includingthe moving object of interest, (3) determining for each object,including the moving object of interest, in the two-dimensional virtualdigital representation an accuracy of actual pointing and in determiningthe accuracy of pointing according to substep (D)(3) using standarddeviations transmitted at step (B) caused by the mobile device ingenerating the mobile device geodetic position and pointing directionand the distance estimate for each object determined at substep (D)(2),(4) defining for each object, including the moving object of interest,in the two-dimensional virtual digital representation an optimalpointing location relating to a visible part of each object, includingthe moving object of interest, (5) determining for each object,including the moving object of interest, in the two-dimensional virtualdigital representation a probability surface for optimal pointing fromthe mobile device geodetic position applying the definition for eachobject defined at substep (D)(4), (6) determining for each object,including the moving object of interest, in the two-dimensional virtualdigital representation, a numeric correspondence between actual pointingaccuracy and optimal pointing location by comparing a standard deviationfor the actual pointing accuracy determined at substep (D)(3) and theprobability surface for optimal pointing determined at substep (D)(5),(7) selecting as the object of interest the object with the highestnumeric correspondence determined at substep (D)(6), (8) updating aposition and direction of movement of each object, including the movingobject of interest, in the two-dimensional virtual digitalrepresentation, and (9) the system server communicating to the mobiledevice an identity of the moving object of interest.
 2. The method asrecited in claim 1, wherein the mobile device and the system servercommunicate wired and wirelessly.
 3. The method as recited in claim 1,wherein the mobile device geodetic position includes a positionaccording to a latitude, longitude, elevation, pitch, and yaw of themobile device.
 4. The method as recited in claim 1, wherein the standarddeviations relating to generating the mobile device geodetic positionand pointing direction, include error propagation caused by mobiledevice sensors.
 5. The method as recited in claim 4, wherein errorpropagations includes geodetic error propagation.
 6. The method asrecited in claim 1, wherein the distance estimate for each object,including the moving object of interest, includes a distance estimatefrom the mobile device geodetic position to a center of gravity of eachobject, including the moving object of interest.
 7. The method asrecited in claim 1, wherein a visible part of each object, including themoving object of interest, includes a visible part from the mobiledevice.
 8. The method as recited in claim 1, wherein determining aprobability surface for each object, including the moving object ofinterest, includes determining a probability surface according to anoutline of each object, including the object of interest, from themobile device geodetic position.
 9. The method as recited in claim 1,wherein substep (D)(6) further includes determining a standard deviationfor the actual pointing accuracy for each object, including the movingobject of interest, in the two-dimensional virtual digitalrepresentation.
 10. The method as recited in claim 9, wherein substep(D)(5) further includes determining for the probability surface foroptional pointing a standard deviation for the optimal pointing locationfor each object, including the moving object of interest, in thetwo-dimensional virtual digital representation.
 11. The method asrecited in claim 10, wherein comparing at substep (D)(6) includescomparing the standard deviation for the actual pointing accuracydetermined at substep (D)(3) and the standard deviation for the optimalpointing location using a t-test.
 12. The method as recited in claim 11,wherein the numeric correspondence for each object, including the movingobject of interest, includes a numeric correspondence of the overlap ofthe standard deviation for the actual pointing accuracy determined atsubstep (D)(3) and the standard deviation for the optimal pointinglocation.
 13. The method as recited in claim 12, wherein numericcorrespondence includes at least a lateral correspondence between thestandard deviation for the actual pointing accuracy determined atsubstep (D)(3) and the standard deviation for optimal pointing locationaccording to an x-axis in the two-dimensional virtual digitalrepresentation including the optimal point location and the actualpointing location.
 14. The method as recited in claim 12, whereinnumeric correspondence includes at least a vertical correspondencebetween the standard deviation for the actual pointing accuracydetermined at substep (D)(3) and the standard deviation for the optimalpointing location according to a y-axis in the two-dimensional virtualdigital representation including the optimal point location and theactual pointing location.
 15. The method as recited in claim 1, whereinthe mobile device includes a handheld device that includes capabilitiesof a pointing device.
 16. A system for determining a likelihood a movingobject in a real-world scene is an object being pointed at by a mobiledevice that includes capabilities of a pointing device, comprising: amobile device that includes the capabilities of a pointing device forgenerating a geodetic position and pointing direction for the mobiledevice, and transmitting the mobile device geodetic position andpointing direction, and standard deviations relating to generating themobile device geodetic position and pointing direction to a systemserver; and a system server further comprising, a mapping module forreceiving and processing a three-dimensional representation of anenvironment that contains a real-world scene with objects, including themoving object of interest, a scene generator module that connects to themapping module and receives an output from the mapping module thatgenerates a two-dimensional virtual digital representation of thereal-world scene including objects, including the moving object ofinterest, and determines for each object, including the moving object ofinterest, in the two-dimensional virtual digital representation adistance estimate from the mobile device geodetic position to each suchmoving object, including the moving object of interest, a pointingaccuracy module that determines for each object, including the movingobject of interest, in the two-dimensional virtual digitalrepresentation an accuracy of actual pointing and in determiningaccuracy of pointing including the standard deviations transmitted fromthe mobile device caused by the mobile device in generating the mobiledevice geodetic position and pointing direction, and the distanceestimate, optimal pointing module that defines for each object,including the moving object interest, in the two-dimensional virtualdigital representation an optimal pointing location relating to avisible part of each object, including the moving object of interest,and for each object, including the moving object of interest, in thetwo-dimensional virtual digital representation a probability surface foroptimal pointing from the mobile device geodetic position to suchobjects, including the moving object of interest, an updater forupdating the position and/or direction of movement of each object ofinterest, including the moving object of interest, in thetwo-dimensional virtual digital representation in real time, and acomparison module that determines for each object, including the movingobject of interest, in the two-dimensional virtual digitalrepresentation, a numeric correspondence between actual pointingaccuracy and optimal pointing location by comparing a standard deviationfor actual pointing accuracy and a probability surface for optimalpointing for each such object, including the moving object of interest,and that selects as the object of interest the object with the highestnumeric correspondence according to a comparison by the comparisonmodule, and a communications module that communicates with the mobiledevice with the communications including transmitting an identity of themoving object of interest to the mobile device.
 17. The system asrecited in claim 15, wherein the mobile device and system servercommunicate wired or wirelessly.
 18. The system as recited in claim 15,wherein the mobile device includes sensors for determining mobile devicegeodetic location and pointing direction.
 19. The system as recited inclaim 18, wherein pointing device sensors cause mobile device geodeticposition errors.
 20. The system as recited in claim 19, when mobiledevice geodetic position includes a position according to a latitude,longitude, elevation, pitch, and yaw of the pointing device.
 21. Thesystem as recited in claim 15, wherein the comparison module includes atesting module for testing comparisons using a t-test.
 22. The system asrecited in claim 16, wherein the mobile device includes a handhelddevice that includes capabilities of a pointing device.